feat:prover-db-indexer- add OCW consumer that drains and forwards events#192
Merged
Conversation
…ents Builds on the HTTP+protobuf client from the previous PR; wires it into a Substrate offchain worker that: - Acquires an OCW storage lock (deadline configurable via `Config::OcwLockDeadlineMs`) so overlapping rounds don't double-submit. - Reads the indexer URL from offchain local storage (key in `sxt-core`) and skips silently if unset, so non-indexer nodes pay nothing. - Asks the server for its last checkpoint sequence number — the server is the sole source of truth, no local cursor. - Walks blocks `cursor+1..=tip` capped by `Config::MaxBlocksPerInvocation`, reads the per-block high-water-mark, probes `key_for_event(block, 0..=hwm)`, forwards each present payload via the HTTP client, and deletes the consumed offchain entries before checkpointing. Other notable bits: - `ConsumerError` Snafu enum keeps the consumer's error stack typed end-to-end; the offchain_worker hook formats the error with its Display impl. - Tests use a `TestOffchainExt` + recording-HTTP transport in the mock to exercise the full forward path, multi-block cursors, partial blocks, lock contention, and resumption from a server checkpoint. - `ROW_NUMBER_COLUMN_NAME` is the indexer's dedup-key column; lifted to `pub` in `commitment-sql` so the consumer can pass it through to `create_table` without duplicating the constant. - `MaxBlocksPerInvocation` (100) and `OcwLockDeadlineMs` (120_000) are exposed as `#[pallet::constant]` items so other chains can override them and so they show up in metadata for polkadot.js.
…st imports, reuse from_str_unchecked
- `[dev-dependencies]` only needs `parking_lot` (for `RwLock` in the
offchain-state shared handle the upstream `TestOffchainExt::new` hands
back). The redundant `polkadot-sdk` dev-dep block was already covered
by `[dependencies]` in test builds, and none of `pallet-tables` /
`pallet-indexing` / `pallet-system-tables` / `pallet-zkpay` /
`pallet-permissions` / `pallet-commitments` /
`proof-of-sql-commitment-map` are referenced by the mock or tests.
Visibly speeds up the test-build dep graph.
- Hoist the `use polkadot_sdk::sp_runtime::offchain::{...}` statements
out of `Pallet::run_consumer` and out of the `ocw_skips_when_lock_is_held`
test so the module's dependencies are visible at a glance.
- Drop the local `table_id` helper in tests in favor of
`TableIdentifier::from_str_unchecked` from sxt-core (note the arg
order: `from_str_unchecked(name, namespace)`).
1.69.0Bug Fixes
Features
|
tlovell-sxt
approved these changes
May 27, 2026
tlovell-sxt
left a comment
tlovell-sxt
left a comment
Contributor
There was a problem hiding this comment.
LGTM thanks for all the iteration on this
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Final PR in the prover-db-indexer stack: wires up the offchain worker that consumes what PR 2 (#187) writes to the offchain DB and POSTs it through the HTTP+protobuf client from PR 3 (#190).
Behavior
offchain_workerfor every imported block; without serialization, two concurrent rounds would read the same server checkpoint and double-submitcreate_table/drop_table/put_batches.StorageLock<Time>with a configurable deadline (Config::OcwLockDeadlineMs, default 120 s) makes overlapping invocations no-ops.PROVER_DB_URL_KEY); skips silently if unset, so non-indexer nodes pay nothing.OCW retries.
cursor+1..=tipcapped atConfig::MaxBlocksPerInvocation(default 100 — ~100× realtime at 6 s block time). Lower values bound per-round cost; higher values close a wider gapfaster after downtime.
key_for_event(block, 0..=hwm)in extrinsic order, POSTs each present payload via the HTTP client, deletes the consumed offchain entries,then checkpoints the block on the server (even if it had zero captures, so the server's view doesn't lag empty blocks).
Notable implementation details
ConsumerErrorSnafu enum keeps the consumer stack typed end-to-end: eachhttp_client::Erroris wrapped with the failing operation (CreateTable/DropTable/PutBatches/Checkpoint) so the operatorlog line names both. The OCW hook formats it with its
Displayimpl and emits viapolkadot_sdk::sp_tracing::error!.MaxBlocksPerInvocationandOcwLockDeadlineMsare#[pallet::constant] type Get<u64>items so other chains can override them and so they show up in the pallet's metadata (visible in polkadot.js UI).Wired with
ConstU64<100>/ConstU64<120_000>inruntime/src/lib.rs.commitment_sql::ROW_NUMBER_COLUMN_NAMEis reused rather than redeclared — the indexer's dedup-key column is the sameMETA_ROW_NUMBERvalue the schema rewriter injects. Pub'd incommitment-sqlso thepallet doesn't duplicate the constant.
TestOffchainExt+ a recording HTTP transport in the mock to exercise the full forward path: golden path, multi-block cursors, partial blocks (high-water-mark with gaps in extrinsic indices),empty blocks, lock contention, and resumption from a server checkpoint mid-stream. 9 tests, all green.
parking_lot(for theRwLockupstream'sTestOffchainExt::newhands back). The mock + tests don't reference any other pallet.Test plan
cargo test -p pallet-prover-db-indexer— 9 OCW testscargo check --workspacecleancargo fclean (workspace formatter)--prover-db-url http://127.0.0.1:<port>against the prb-service indexer atsxtdb/prb-service; create a table viapallet-tables::create_tables, submit data, watchthe OCW round forward
create_table+put_batchesto the indexer's v1 HTTP API.Stacked PRs
This is PR 3 of 3 in the prover-db-indexer stack:
EventCaptureproducer